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1.  The  Predictive  Performance  Optimizer 

Building  on  more  than  a  century  of  research  on  human 
memory  and  performance,  the  Predictive  Performance 
Optimizer  (PPO)  is  a  state-of-the-art  cognitive  tool  to 
help  decision-makers,  instructors,  and  learners  of  all 
types  to  assess  current  performance  and  predict  future 
performance  by  capturing  the  dynamics  of  human 
learning  with  basic  cognitive  science  principles. 

The  PPO  is  a  user-friendly  software  tool  that  can  track 
performance  over  the  course  of  a  learner’s  training 
history  for  virtually  any  quantitative  measure  of 
performance.  It  generates  performance  predictions  at 
specified  future  points  in  time,  and  allows  users  to 
visually  and  graphically  assess  and  compare  the  impact 
of  potential  future  training  regimens.  The  PPO 
accomplishes  this  by  utilizing  a  mathematical  model  for 
performance  prediction  (shown  in  Equation  1.1  below) 
inspired  by  the  General  Performance  Equation 
(Anderson  &  Schunn,  2000). 

Performance  =  S  *  St  *  AT  *  Td  (Equation  1.1). 

It  comprises  three  main  parts:  the  power  law  of  learning 
(M),  the  power  law  of  forgetting  ( Td ),  and  a  stability 
term  (St)  which  captures  the  effects  of  practice  and 
retention  as  they  are  spaced  over  time.  The  combination 
of  these  terms,  along  with  a  scaling  factor  (S),  produces 
point  predictions  of  future  performance  based  on 
mathematical  regularities  in  the  learner’s  historical 
performance  (for  additional  details,  see  Jastrzembski  et 
al.,  2009). 

A  major  intended  use  of  the  PPO  is  to  provide  instructors 
and  trainers  with  principled  guidance  concerning  the 


readiness  of  their  trainees.  We  will  now  frame  PPO’s 
practical  relevance  into  a  “just-in-time”  training  refresher 
scenario.  Consider  a  training  manager  attempting  to 
gauge  how  much  training  a  warfighter  must  receive  to 
ensure  performance  at  or  above  a  specified  level  of 
effectiveness  before  he  may  be  deployed.  The  training 
manager  may  load  the  warfighter’s  unique  training 
history  into  PPO  to  generate  point  predictions  of  future 
performance.  The  training  manager  can  then  assess 
whether  adjustments  must  be  made  to  the  future  training 
routine  to  meet  the  desired  training  goals. 

Given  the  variability  in  human  performance,  generation 
of  pure  point  predictions  is  insufficient  in  helping 
training  managers  make  critical  training  decisions.  One 
can  imagine  a  scenario  where  a  point  prediction  is  at  or 
very  close  to  the  effectiveness  standard.  Should  the 
training  regimen  be  deemed  sufficient  in  that  case?  Is 
additional  training  heeded?  Can  we  be  confident  that  the 
performer  will  achieve  that  level  of  effectiveness  at  all? 
It  is  therefore  necessary  to  provide  training  managers 
with  scientifically-principled  estimates  of  risk  around  the 
model’s  point  predictions,  to  better  guide  decisions  that 
have  an  impact  in  the  real-world.  We  now  turn  to  a 
discussion  concerning  how  best  to  compute  a  prediction 
interval  (PI)  around  the  model’s  point  predictions. 

2.  Prediction  Interval  Calculations 

Rather  than  discrete  point  predictions,  Pis  provide  a 
range  of  possible  values  of  future  performance,  and  thus 
offer  the  trainer  a  more  complete  picture  of  what 
outcomes  future  training  regimens  may  possess. 
Identifying  a  method  to  compute  a  principled  PI  for  our 
needs,  however,  is  far  from  straightforward. 
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One  issue  we  face  is  that  we  must  balance  two 
interacting  effects:  on  one  hand,  human  performance 
generally  becomes  less  variable  with  increased  practice; 
on  the  other  hand,  model  predictions  generally  become 
less  certain  with  longer  lead  times.  A  second  issue  is  the 
limited  existing  data  with  which  to  validate  the  model’s 
extended  predictions.  Related  fields,  such  as  economics, 
typically  possess  data  spanning  months  or  years,  but  few 
psychological  studies  examine  data  across  time  scales 
longer  than  a  few  days.  A  third  problem  is  that  there  is 
little  in  the  psychological  literature  which  focuses  on 
predicting  performance  at  future  times,  and  within  that 
research,  the  incorporation  of  Pis  on  future  performance 
is  almost  entirely  absent.  Thus,  we  lack  sufficient 
exemplars  to  directly  apply  any  one  methodology  to  our 
situation,  and  have  turned  to  other  disciplines  (e.g., 
econometrics  and  biostatistics),  whose  application  to  our 
situation  is  less  straightforward,  for  guidance  as  a  result. 
A  final  hurdle  is  maintaining  the  generality  of  the  model. 
The  model  is  intended  to  be  used  for  predicting 
performance  in  a  wide  range  of  areas,  and  thus  a  large 
range  of  dependent  variables.  Accordingly,  any 
methodology  to  compute  Pis  must  not  make 
mathematical  assumptions  that  cannot  be  met  with  most 
measures  of  performance. 

One  method  commonly  used  to  generate  Pis  is  the 
incorporation  of  a  noise  parameter  into  one  or  more  parts 
of  the  model.  In  a  computational  model,  this  can  be 
relatively  straightforward,  and  the  ACT-R  framework 
has  several  extant  noise  parameters  that  can  be  utilized  in 
a  variety  of  situations.  In  our  mathematical 
implementation,  however,  it  is  less  obvious  how  to  add 
in  a  noise  parameter.  As  such,  we  are  evaluating  which 
terms  in  our  mathematical  model  have  a  strong 
theoretical  motivation  to  vary,  and  how  these  terms 
might  interact  with  one  another.  For  example,  the 
learning  rate  and/or  the  forgetting  rate  might  vary  from 
one  training  session  to  the  next  based  on  fluctuations  in 
the  attentiveness  of  the  warfighter  or  variability  in  the 
quality  of  the  information  in  the  briefing  before  the 
training  session  begins.  However,  one  still  has  to 
determine  the  form  and  magnitude  of  the  distribution 
from  which  to  sample  the  noise.  For  this,  we  are 
investigating  measures  of  variability  in  model  fits  to 
observed  data  that  may  be  used  to  estimate  the  variability 
expected  in  future  data. 

The  resulting  Pis  from  this  method,  or  any  similar 
method,  on  predicted  future  performance  provide  an 
important  tool  for  trainers  and  decision-makers  by 
presenting  a  range  of  likely  values  for  future 
performance.  In  our  warfighter  scenario,  the  training 
manager  may  decide  to  adopt  a  conservative  criterion 
and  use  the  worst  likely  performance  shown  by  the  Pis  as 
a  guide  to  impact  future  training  needs.  Such  a  criterion 
would  ensure  that  the  warfighter  is  most  likely  to 


actually  perform  at  or  above  the  desired  level  of 
effectiveness. 

3.  Summary 

The  question  of  how  to  properly  calculate  Pis  for  a 
mathematical  model  of  performance  and  learning  is  a 
challenging  one.  The  existing  psychological  literature 
offers  little  insight.  We  are,  however,  investigating  a 
number  of  promising  methods  from  related  fields. 
Specifically,  implementing  noise  in  the  model  to 
generate  variability  is  one  of  several  promising 
possibilities.  The  development  of  an  elegant  method  for 
calculating  Pis  for  psychological  performance  data 
would  hopefully  encourage  widespread  use  of  such 
intervals  as  opposed  to  simple  point  predictions  which 
inherently  have  unspecified  certainty  in  their  precise 
value.  Our  poster  will  present  results  from  our  ongoing 
explorations  of  these  methods. 
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